Skip to main content

Reply to this post | Go Back
View Post [edit]

Poster: Sinkevich Date: Oct 5, 2013 5:59am
Forum: web Subject: Please, check my robots.txt

Good day.
I have a forum.

I want that ia_archiver crawls the index page and pages of partitions of forum.

But I want that ia_archiver will delete all pages of topics of forum from base.

Please, check my robots.txt.
Is it true?

User-agent: ia_archiver
Disallow: /topic